Why The Future Doesn't Need Us 



By Bill Joy 




Our most powerful 21st-century technologies - robotics, genetic engineering, and nanotech - are 
threatening to make humans an endangered species. 

From the moment I became involved in the creation of new technologies, their ethical dimensions have 
concerned me, but it was only in the autumn of 1998 that 1 became anxiously aware of how great are 
the dangers facing us in the 21st century. 1 can date the onset of my unease to the day I met Ray 
Kurzweil, the deservedly famous inventor of the first reading machine for the blind and many other 
amazing things. 

Ray and 1 were both speakers at George Gilder's Telecosm conference, and I encountered him by 
chance in the bar of the hotel after both our sessions were over. I was sitting with John Searle, a 
Berkeley philosopher who studies consciousness. While we were talking, Ray approached and a 
conversation began, the subject of which haunts me to this day. 

I had missed Ray's talk and the subsequent panel that Ray and John had been on, and they now picked 
right up where they'd left off, with Ray saying that the rate of improvement of technology was going to 
accelerate and that we were going to become robots or fuse with robots or something like that, and 
John countering that this couldn't happen, because the robots couldn't be conscious. 

While I had heard such talk before, I had always felt sentient robots were in the realm of science 
fiction. But now, from someone I respected, I was hearing a strong argument that they were a near-term 
possibility. I was taken aback, especially given Ray's proven ability to imagine and create the future. I 



already knew that new technologies like genetic engineering and nanotechnology were giving us the 
power to remake the world, but a realistic and imminent scenario for intelligent robots surprised me. 

It's easy to get jaded about such breakthroughs. We hear in the news almost every day of some kind of 
technological or scientific advance. Yet this was no ordinary prediction. In the hotel bar, Ray gave me a 
partial preprint of his then- forthcoming hookThe Age of Spiritual Machines, which outlined a Utopia he 
foresaw - one in which humans gained near immortality by becoming one with robotic technology. On 
reading it, my sense of unease only intensified; I felt sure he had to be understating the dangers, 
understating the probability of a bad outcome along this path. 

I found myself most troubled by a passage detailing adystopian scenario: 

THE NEW LUDDITE CHALLENGE 

First let us postulate that the computer scientists succeed in developing intelligent machines that can do 
all things better than human beings can do them. In that case presumably all work will be done by vast, 
highly organized systems of machines and no human effort will be necessary. Either of two cases might 
occur. The machines might be permitted to make all of their own decisions without human oversight, or 
else human control over the machines might be retained. 

If the machines are permitted to make all their own decisions, we can't make any conjectures as to the 
results, because it is impossible to guess how such machines might behave. We only point out that the 
fate of the human race would be at the mercy of the machines. It might be argued that the human race 
would never be foolish enough to hand over all the power to the machines. But we are suggesting 
neither that the human race would voluntarily turn power over to the machines nor that the machines 
would willfully seize power. What we do suggest is that the human race might easily permit itself to 
drift into a position of such dependence on the machines that it would have no practical choice but to 
accept all of the machines' decisions. As society and the problems that face it become more and more 
complex and machines become more and more intelligent, people will let machines make more of their 
decisions for them, simply because machine-made decisions will bring better results than man-made 
ones. Eventually a stage may be reached at which the decisions necessary to keep the system running 
will be so complex that human beings will be incapable of making them intelligently. At that stage the 



machines will be in effective control. People won't be able to just turn the machines off, because they 
will be so dependent on them that turning them off would amount to suicide. 

On the other hand it is possible that human control over the machines may be retained. In that case the 
average man may have control over certain private machines of his own, such as his car or his personal 
computer, but control over large systems of machines will be in the hands of a tiny elite - just as it is 
today, but with two differences. Due to improved techniques the elite will have greater control over the 
masses; and because human work will no longer be necessary the masses will be superfluous, a useless 
burden on the system. If the elite is ruthless they may simply decide to exterminate the mass of 
humanity. If they are humane they may use propaganda or other psychological or biological techniques 
to reduce the birth rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if 
the elite consists of soft-hearted liberals, they may decide to play the role of good shepherds to the rest 
of the human race. They will see to it that everyone's physical needs are satisfied, that all children are 
raised under psychologically hygienic conditions, that everyone has a wholesome hobby to keep him 
busy, and that anyone who may become dissatisfied undergoes "treatment" to cure his "problem." Of 
course, life will be so purposeless that people will have to be biologically or psychologically 
engineered either to remove their need for the power process or make them "sublimate" their drive for 
power into some harmless hobby. These engineered human beings may be happy in such a society, but 
they will most certainly not be free. They will have been reduced to the status of domestic animals.! 



1 The passage Kurzweil quotes is from Kaczynski's Unabomber Manifesto, which was published 
jointly, under duress, hyThe New York Times and The Washington Post to attempt to bring his campaign 
of terror to an end. I agree with David Gelemter, who said about their decision: 

"It was a tough call for the newspapers. To say yes would be giving in to terrorism, and for all they 
knew he was lying anyway. On the other hand, to say yes might stop the killing. There was also a 
chance that someone would read the tract and get a hunch about the author; and that is exactly what 
happened. The suspect's brother read it, and it rang a bell. 

"I would have told them not to publish. I'm glad they didn't ask me. I guess." 

(Drawing Life: Surviving the Unabomber. Free Press, 1997: 120.) 

In the book, you don't discover until you turn the page that the author of this passage is Theodore 



Kaczynski - the Unabomber. I am no apologist for Kaczynski. His bombs killed three people during a 
17-year terror campaign and wounded many others. One of his bombs gravely injured my friend David 
Gelemter, one of the most brilliant and visionary computer scientists of our time. Like many of my 
colleagues, I felt that I could easily have been the Unabomber's next target. 

Kaczynski's actions were murderous and, in my view, criminally insane. He is clearly a Luddite, but 
simply saying this does not dismiss his argument; as difficult as it is for me to acknowledge, I saw 
some merit in the reasoning in this single passage. I felt compelled to confront it. 

Kaczynski's dystopian vision describes unintended consequences, a well-known problem with the 
design and use of technology, and one that is clearly related to Murphy's law - "Anything that can go 
wrong, will." (Actually, this is Finagle's law, which in itself shows that Finagle was right.) Our overuse 
of antibiotics has led to what may be the biggest such problem so far: the emergence of antibiotic- 
resistant and much more dangerous bacteria. Similar things happened when attempts to eliminate 
malarial mosquitoes using DDT caused them to acquire DDT resistance; malarial parasites likewise 
acquired multi-drug-resistant genes. 2 

The cause of many such surprises seems clear: The systems involved are complex, involving 
interaction among and feedback between many parts. Any changes to such a system will cascade in 
ways that are difficult to predict; this is especially true when human actions are involved. 

I started showing friends the Kaczynski quote iwmThe Age of Spiritual Machines; I would hand them 
Kurzweil's book, let them read the quote, and then watch their reaction as they discovered who had 
written it. At around the same time, I found Hans Moravec's hookRobot: Mere Machine to 
Transcendent Mind. Moravec is one of the leaders in robotics research, and was a founder of the 
world's largest robotics research program, at Carnegie Mellon University.i?oi»o? gave me more material 
to try out on my friends - material surprisingly supportive of Kaczynski's argument. For example: 

The Short Run (Early 2000s) 

Biological species almost never survive encounters with superior competitors. Ten million years ago. 
South and North America were separated by a sunken Panama isthmus. South America, like Australia 
today, was populated by marsupial mammals, including pouched equivalents of rats, deers, and tigers. 
When the isthmus connecting North and South America rose, it took only a few thousand years for the 
northern placental species, with slightly more effective metabolisms and reproductive and nervous 
systems, to displace and eliminate almost all the southern marsupials. 

In a completely free marketplace, superior robots would surely affect humans as North American 
placentals affected South American marsupials (and as humans have affected countless species). 
Robotic industries would compete vigorously among themselves for matter, energy, and space, 
incidentally driving their price beyond human reach. Unable to afford the necessities of life, biological 
humans would be squeezed out of existence. 

There is probably some breathing room, because we do not live in a completely free marketplace. 
Government coerces nonmarket behavior, especially by collecting taxes. Judiciously applied, 
governmental coercion could support human populations in high style on the fruits of robot labor, 
perhaps for a long while. 

A textbook dystopia - and Moravec is just getting wound up. He goes on to discuss how our main job in 
the 21st century will be "ensuring continued cooperation from the robot industries" by passing laws 
decreeing that they be "nice,"3 and to describe how seriously dangerous a human can be "once 
transformed into an unbounded superintelligent robot." Moravec's view is that the robots will 
eventually succeed us - that humans clearly face extinction. 



I decided it was time to talk to my friend Damiy Hillis. Damiy became famous as the cofounder of 
Thinking Machines Corporation, which built a very powerful parallel supercomputer. Despite my 
current job title of Chief Scientist at Sun Microsystems, I am more a computer architect than a scientist, 
and I respect Danny's knowledge of the information and physical sciences more than that of any other 
single person I know. Danny is also a highly regarded futurist who thinks long-term - four years ago he 
started the Long Now Foundation, which is building a clock designed to last 10,000 years, in an 
attempt to draw attention to the pitifully short attention span of our society. (See " Test of Time ." Wired 
8.03, page 78.) 

So I flew to Los Angeles for the express purpose of having dinner with Danny and his wife, Pati. I went 
through my now-familiar routine, trotting out the ideas and passages that I found so disturbing. Danny's 
answer - directed specifically at Kurzweil's scenario of humans merging with robots - came swiftly, and 
quite surprised me. He said, simply, that the changes would come gradually, and that we would get 
used to them. 

But I guess I wasn't totally surprised. I had seen a quote from Danny in Kurzweil's book in which he 
said, "I'm as fond of my body as anyone, but if I can be 200 with a body of silicon, I'll take it." It 
seemed that he was at peace with this process and its attendant risks, while I was not. 

While talking and thinking about Kurzweil, Kaczynski, and Moravec, I suddenly remembered a novel I 
had read almost 20 years ago -The White Plague, by Frank Herbert - in which a molecular biologist is 
driven insane by the senseless murder of his family. To seek revenge he constructs and disseminates a 
new and highly contagious plague that kills widely but selectively. (We're lucky Kaczynski was a 
mathematician, not a molecular biologist.) I was also reminded of the Borg of^tor Trek, a hive of partly 
biological, partly robotic creatures with a strong destructive streak. Borg-like disasters are a staple of 
science fiction, so why hadn't I been more concerned about such robotic dystopias earlier? Why weren't 
other people more concerned about these nightmarish scenarios? 



2 Garrett, Laurie. T/ze Coming Plague: Newly Emerging Diseases in a World Out of Balance. Penguin, 
1994: 47-52,414,419, 452. 

3 Isaac Asimov described what became the most famous view of ethical rules for robot behavior in his 
book/. Robot in 1950, in his Three Laws of Robotics: 1. A robot may not injure a human being, or, 
through inaction, allow a human being to come to harm. 2. A robot must obey the orders given it by 
human beings, except where such orders would conflict with the First Law. 3. A robot must protect its 
own existence, as long as such protection does not conflict with the First or Second Law. 

Part of the answer certainly lies in our attitude toward the new - in our bias toward instant familiarity 
and unquestioning acceptance. Accustomed to living with almost routine scientific breakthroughs, we 
have yet to come to terms with the fact that the most compelling 2 1 st-century technologies - robotics, 
genetic engineering, and nanotechnology - pose a different threat than the technologies that have come 
before. Specifically, robots, engineered organisms, and nanobots share a dangerous amplifying factor: 
They can self-replicate. A bomb is blown up only once - but one bot can become many, and quickly get 
out of control. 

Much of my work over the past 25 years has been on computer networking, where the sending and 
receiving of messages creates the opportunity for out-of-control replication. But while replication in a 
computer or a computer network can be a nuisance, at worst it disables a machine or takes down a 
network or network service. Uncontrolled self-replication in these newer technologies runs a much 
greater risk: a risk of substantial damage in the physical world. 



Each of these technologies also offers untold promise: The vision of near immortality that Kurzweil 
sees in his robot dreams drives us forward; genetic engineering may soon provide treatments, if not 
outright cures, for most diseases; and nanotechnology and nanomedicine can address yet more ills. 
Together they could significantly extend our average life span and improve the quality of our lives. Yet, 
with each of these technologies, a sequence of small, individually sensible advances leads to an 
accumulation of great power and, concomitantly, great danger. 

What was different in the 20th century? Certainly, the technologies underlying the weapons of mass 
destruction (WMD) - nuclear, biological, and chemical (NBC) - were powerful, and the weapons an 
enormous threat. But building nuclear weapons required, at least for a time, access to both rare - 
indeed, effectively unavailable - raw materials and highly protected information; biological and 
chemical weapons programs also tended to require large-scale activities. 

The 21st-century technologies - genetics, nanotechnology, and robotics (GNR) - are so powerful that 
they can spawn whole new classes of accidents and abuses. Most dangerously, for the first time, these 
accidents and abuses are widely within the reach of individuals or small groups. They will not require 
large facilities or rare raw materials. Knowledge alone will enable the use of them. 

Thus we have the possibiUty not just of weapons of mass destruction but of knowledge-enabled mass 
destruction (KMD), this destructiveness hugely amplified by the power of self-replication. 

I think it is no exaggeration to say we are on the cusp of the further perfection of extreme evil, an evil 
whose possibility spreads well beyond that which weapons of mass destruction bequeathed to the 
nation-states, on to a surprising and terrible empowerment of extreme individuals. 

Nothing about the way I got involved with computers suggested to me that I was going to be facing 
these kinds of issues. 

My life has been driven by a deep need to ask questions and find answers. When I was 3, 1 was already 
reading, so my father took me to the elementary school, where I sat on the principal's lap and read him 
a story. 1 started school early, later skipped a grade, and escaped into books - I was incredibly motivated 
to learn. I asked lots of questions, often driving adults to distraction. 

As a teenager I was very interested in science and technology. I wanted to be a ham radio operator but 
didn't have the money to buy the equipment. Ham radio was the Internet of its time: very addictive, and 
quite solitary. Money issues aside, my mother put her foot down - 1 was not to be a ham; I was 
antisocial enough aheady. 

I may not have had many close friends, but I was awash in ideas. By high school, I had discovered the 
great science fiction writers. I remember especially Heinlein's//ave Spacesuit Will Travel and Asimov's 
/, Robot, with its Three Laws of Robotics. I was enchanted by the descriptions of space travel, and 
wanted to have a telescope to look at the stars; since I had no money to buy or make one, I checked 
books on telescope-making out of the library and read about making them instead. I soared in my 
imagination. 

Thursday nights my parents went bowling, and we kids stayed home alone. It was the night of Gene 
Roddenberry's original Star Trek, and the program made a big impression on me. I came to accept its 
notion that humans had a future in space. Western-style, with big heroes and adventures. Roddenberry's 
vision of the centuries to come was one with strong moral values, embodied in codes like the Prime 
Directive: to not interfere in the development of less technologically advanced civilizations. This had 
an incredible appeal to me; ethical humans, not robots, dominated this future, and I took Roddenberry's 
dream as part of my own. 



I excelled in mathematics in high school, and when I went to the University of Michigan as an 
undergraduate engineering student I took the advanced curriculum of the mathematics majors. Solving 
math problems was an exciting challenge, but when I discovered computers I found something much 
more interesting: a machine into which you could put a program that attempted to solve a problem, 
after which the machine quickly checked the solution. The computer had a clear notion of correct and 
incorrect, true and false. Were my ideas correct? The machine could tell me. This was very seductive. 

I was lucky enough to get a job programming early supercomputers and discovered the amazing power 
of large machines to numerically simulate advanced designs. When I went to graduate school at UC 
Berkeley in the mid-1970s, I started staying up late, often all night, inventing new worlds inside the 
machines. Solving problems. Writing the code that argued so strongly to be written. 

InThe Agony and the Ecstasy, Irving Stone's biographical novel of Michelangelo, Stone described 
vividly how Michelangelo released the statues from the stone, "breaking the marble spell," carving 
from the images in his mind.4 In my most ecstatic moments, the software in the computer emerged in 
the same way. Once I had imagined it in my mind I felt that it was already there in the machine, waiting 
to be released. Staying up all night seemed a small price to pay to free it - to give the ideas concrete 
form. 



4 Michelangelo wrote a sonnet that begins: 

Non ha /' ottimo artista alcun concetto 
Ch ' un marmo solo in se non circonscriva 
Col suo soverchio; e solo a quello arriva 
La man che ubbidisce all' intelleto. 

Stone translates this as: 

The best of artists hath no thought to show 
which the rough stone in its superfluous shell 
doth not include; to break the marble spell 
is all the hand that serves the brain can do. 

Stone describes the process: "He was not working from his drawings or clay models; they had all been 
put away. He was carving from the images in his mind. His eyes and hands knew where every line, 
curve, mass must emerge, and at what depth in the heart of the stone to create the low relief." 

{The Agony and the Ecstasy. Doubleday, 1961: 6, 144.) 

After a few years at Berkeley I started to send out some of the software I had written - an instructional 
Pascal system, Unix utilities, and a text editor called vi (which is still, to my surprise, widely used more 
than 20 years later) - to others who had similar small PDP-1 1 and VAX minicomputers. These 
adventures in software eventually turned into the Berkeley version of the Unix operating system, which 
became a personal "success disaster" - so many people wanted it that I never finished my PhD. Instead 
I got a job working for Darpa putting Berkeley Unix on the Internet and fixing it to be reliable and to 
run large research applications well. This was all great fun and very rewarding. And, frankly, I saw no 
robots here, or anywhere near. 

Still, by the early 1980s, I was drowning. The Unix releases were very successful, and my little project 
of one soon had money and some staff, but the problem at Berkeley was always office space rather than 
money - there wasn't room for the help the project needed, so when the other founders of Sun 
Microsystems showed up I jumped at the chance to join them. At Sun, the long hours continued into the 



early days of workstations and personal computers, and I have enjoyed participating in the creation of 
advanced microprocessor technologies and Internet technologies such as Java and Jini. 

From all this, I trust it is clear that I am not a Luddite. I have always, rather, had a strong belief in the 
value of the scientific search for truth and in the ability of great engineering to bring material progress. 
The Industrial Revolution has immeasurably improved everyone's life over the last couple hundred 
years, and I always expected my career to involve the building of worthwhile solutions to real 
problems, one problem at a time. 

I have not been disappointed. My work has had more impact than I had ever hoped for and has been 
more widely used than I could have reasonably expected. I have spent the last 20 years still trying to 
figure out how to make computers as reliable as I want them to be (they are not nearly there yet) and 
how to make them simple to use (a goal that has met with even less relative success). Despite some 
progress, the problems that remain seem even more daunting. 

But while I was aware of the moral dilemmas surrounding technology's consequences in fields like 
weapons research, I did not expect that I would confront such issues in my own field, or at least not so 
soon. 

Perhaps it is always hard to see the bigger impact while you are in the vortex of a change. Failing to 

understand the consequences of our inventions while we are in the rapture of discovery and innovation 
seems to be a common fault of scientists and technologists; we have long been driven by the 
overarching desire to know that is the nature of science's quest, not stopping to notice that the progress 
to newer and more powerful technologies can take on a life of its own. 

I have long realized that the big advances in information technology come not from the work of 
computer scientists, computer architects, or electrical engineers, but from that of physical scientists. 
The physicists Stephen Wolfram and Brosl Hasslacher introduced me, in the early 1980s, to chaos 
theory and nonlinear systems. In the 1990s, I learned about complex systems from conversations with 
Danny Hillis, the biologist Stuart Kauffrnan, the Nobel-laureate physicist Murray Gell-Mann, and 
others. Most recently, Hasslacher and the electrical engineer and device physicist Mark Reed have been 
giving me insight into the incredible possibilities of molecular electronics. 

In my own work, as codesigner of three microprocessor architectures - SPARC, picoJava, and MAJC - 
and as the designer of several implementations thereof, I've been afforded a deep and firsthand 
acquaintance with Moore's law. For decades, Moore's law has correctly predicted the exponential rate 
of improvement of semiconductor technology. Until last year I believed that the rate of advances 
predicted by Moore's law might continue only until roughly 2010, when some physical limits would 
begin to be reached. It was not obvious to me that a new technology would arrive in time to keep 
performance advancing smoothly. 

But because of the recent rapid and radical progress in molecular electronics - where individual atoms 
and molecules replace lithographically drawn transistors - and related nanoscale technologies, we 
should be able to meet or exceed the Moore's law rate of progress for another 30 years. By 2030, we 
are likely to be able to build machines, in quantity, a million times as powerful as the personal 
computers of today - sufficient to implement the dreams of Kurzweil and Moravec. 

As this enormous computing power is combined with the manipulative advances of the physical 
sciences and the new, deep understandings in genetics, enormous transformative power is being 
unleashed. These combinations open up the opportunity to completely redesign the world, for better or 
worse: The replicating and evolving processes that have been confined to the natural world are about to 
become realms of human endeavor. 



In designing software and microprocessors, I have never had the feehng that I was designing an 
intelUgent machine. The software and hardware is so fragile and the capabiUties of the machine to 
"think" so clearly absent that, even as a possibihty, this has always seemed very far in the future. 

But now, with the prospect of human-level computing power in about 30 years, a new idea suggests 
itself: that I may be working to create tools which will enable the construction of the technology that 
may replace our species. How do I feel about this? Very uncomfortable. Having struggled my entire 
career to build reliable software systems, it seems to me more than likely that this future will not work 
out as well as some people may imagine. My personal experience suggests we tend to overestimate our 
design abilities. 

Given the incredible power of these new technologies, shouldn't we be asking how we can best coexist 
with them? And if our own extinction is a likely, or even possible, outcome of our technological 
development, shouldn't we proceed with great caution? 

The dream of robotics is, first, that intelligent machines can do our work for us, allowing us lives of 
leisure, restoring us to Eden. Yet in his history of such ideas,Z)arwm Among the Machines, George 
Dyson warns: "In the game of life and evolution there are three players at the table: human beings, 
nature, and machines. I am firmly on the side of nature. But nature, I suspect, is on the side of the 
machines." As we have seen, Moravec agrees, believing we may well not survive the encounter with 
the superior robot species. 

How soon could such an intelligent robot be built? The coming advances in computing power seem to 
make it possible by 2030. And once an intelligent robot exists, it is only a small step to a robot species - 
to an intelligent robot that can make evolved copies of itself. 

A second dream of robotics is that we will gradually replace ourselves with our robotic technology, 
achieving near immortality by downloading our consciousnesses; it is this process that Danny Hillis 
thinks we will gradually get used to and that Ray Kurzweil elegantly details mThe Age of Spiritual 
Machines. (We are beginning to see intimations of this in the implantation of computer devices into the 
human body, as illustrated on thecover oiWired 8.02.) 

But if we are downloaded into our technology, what are the chances that we will thereafter be ourselves 
or even human? It seems to me far more likely that a robotic existence would not be like a human one 
in any sense that we understand, that the robots would in no sense be our children, that on this path our 
humanity may well be lost. 

Genetic engineering promises to revolutionize agriculture by increasing crop yields while reducing the 
use of pesticides; to create tens of thousands of novel species of bacteria, plants, viruses, and animals; 
to replace reproduction, or supplement it, with cloning; to create cures for many diseases, increasing 
our life span and our quality of life; and much, much more. We now know with certainty that these 
profound changes in the biological sciences are imminent and will challenge all our notions of what life 
is. 

Technologies such as human cloning have in particular raised our awareness of the profound ethical 
and moral issues we face. If, for example, we were to reengineer ourselves into several separate and 
unequal species using the power of genetic engineering, then we would threaten the notion of equality 
that is the very cornerstone of our democracy. 

Given the incredible power of genetic engineering, it's no surprise that there are significant safety 
issues in its use. My friend Amory Lovins recently cowrote, along with Hunter Lovins, an editorial that 
provides an ecological view of some of these dangers. Among their concerns: that "the new botany 
aligns the development of plants with their economic, not evolutionary, success." (See " A Tale of Two 
Botanies ." page 247.) Amory's long career has been focused on energy and resource efficiency by 



taking a whole-system view of human-made systems; such a whole-system view often finds simple, 
smart solutions to otherwise seemingly difficult problems, and is usefully applied here as well. 

After reading the Lovins' editorial, I saw an op-ed by Gregg Easterbrook inThe New York Times 
(November 19, 1999) about genetically engineered crops, under the headline: "Food for the Future: 
Someday, rice will have built-in vitamin A. Unless the Luddites win." 

Are Amory and Hunter Lovins Luddites? Certainly not. I believe we all would agree that golden rice, 
with its built-in vitamin A, is probably a good thing, if developed with proper care and respect for the 
likely dangers in moving genes across species boundaries. 

Awareness of the dangers inherent in genetic engineering is beginning to grow, as reflected in the 
Lovins' editorial. The general public is aware of, and uneasy about, genetically modified foods, and 
seems to be rejecting the notion that such foods should be permitted to be unlabeled. 

But genetic engineering technology is already very far along. As the Lovins note, the USDA has 
already approved about 50 genetically engineered crops for unlimited release; more than half of the 
world's soybeans and a third of its com now contain genes spliced in from other forms of life. 

While there are many important issues here, my own major concern with genetic engineering is 
narrower: that it gives the power - whether militarily, accidentally, or in a deliberate terrorist act - to 
create a White Plague. 

The many wonders of nanotechnology were first imagined by the Nobel-laureate physicist Richard 
Feynman in a speech he gave in 1959, subsequently published under the title "There's Plenty of Room 
at the Bottom." The book that made a big impression on me, in the mid-'80s, was Eric Drexler' sEngines 
of Creation, in which he described beautifully how manipulation of matter at the atomic level could 
create a Utopian future of abundance, where just about everything could be made cheaply, and almost 
any imaginable disease or physical problem could be solved using nanotechnology and artificial 
intelligences. 

A subsequent hook,Unbounding the Future: The Nanotechnology Revolution, which Drexler cowrote, 
imagines some of the changes that might take place in a world where we had molecular-level 
"assemblers." Assemblers could make possible incredibly low-cost solar power, cures for cancer and 
the common cold by augmentation of the human immune system, essentially complete cleanup of the 
environment, incredibly inexpensive pocket supercomputers - in fact, any product would be 
manufacturable by assemblers at a cost no greater than that of wood - spacefiight more accessible than 
transoceanic travel today, and restoration of extinct species. 

I remember feeling good about nanotechnology after XQ2L6m§,Engines of Creation. As a technologist, it 
gave me a sense of calm - that is, nanotechnology showed us that incredible progress was possible, and 
indeed perhaps inevitable. If nanotechnology was our future, then I didn't feel pressed to solve so many 
problems in the present. I would get to Drexler's Utopian future in due time; I might as well enjoy life 
more in the here and now. It didn't make sense, given his vision, to stay up all night, all the time. 

Drexler's vision also led to a lot of good fun. I would occasionally get to describe the wonders of 
nanotechnology to others who had not heard of it. After teasing them with all the things Drexler 
described I would give a homework assignment of my own: "Use nanotechnology to create a vampire; 
for extra credit create an antidote." 

With these wonders came clear dangers, of which I was acutely aware. As I said at a nanotechnology 
conference in 1989, "We can't simply do our science and not worry about these ethical issues. "5 But my 
subsequent conversations with physicists convinced me that nanotechnology might not even work - or, 
at least, it wouldn't work anytime soon. Shortly thereafter I moved to Colorado, to a skunk works I had 



set up, and the focus of my work shifted to software for the Internet, specifically on ideas that became 
Java and Jini. 

Then, last summer, Brosl Hasslacher told me that nanoscale molecular electronics was now practical. 
This was«ew news, at least to me, and I think to many people - and it radically changed my opinion 
about nanotechnology. It sent me back toEngines of Creation. Rereading Drexler's work after more 
than 10 years, I was dismayed to realize how little I had remembered of its lengthy section called 
"Dangers and Hopes," including a discussion of how nanotechnologies can become "engines of 
destruction." Indeed, in my rereading of this cautionary material today, I am struck by how naive some 
of Drexler's safeguard proposals seem, and how much greater I judge the dangers to be now than even 
he seemed to then. (Having anticipated and described many technical and political problems with 
nanotechnology, Drexler started the Foresight Institute in the late 1980s "to help prepare society for 
anticipated advanced technologies" - most important, nanotechnology.) 

The enabling breakthrough to assemblers seems quite likely within the next 20 years. Molecular 
electronics - the new subfield of nanotechnology where individual molecules are circuit elements - 
should mature quickly and become enormously lucrative within this decade, causing a large 
incremental investment in all nanotechnologies. 

Unfortunately, as with nuclear technology, it is far easier to create destructive uses for nanotechnology 
than constructive ones. Nanotechnology has clear military and terrorist uses, and you need not be 
suicidal to release a massively destructive nanotechnological device - such devices can be built to be 
selectively destructive, affecting, for example, only a certain geographical area or a group of people 
who are genetically distinct. 

An immediate consequence of the Faustian bargain in obtaining the great power of nanotechnology is 
that we run a grave risk - the risk that we might destroy the biosphere on which all life depends. 

As Drexler explained: 

"Plants" with "leaves" no more efficient than today's solar cells could out-compete real plants, 
crowding the biosphere with an inedible foliage. Tough omnivorous "bacteria" could out-compete real 
bacteria: They could spread like blowing pollen, replicate swiftly, and reduce the biosphere to dust in a 
matter of days. Dangerous replicators could easily be too tough, small, and rapidly spreading to stop - 
at least if we make no preparation. We have trouble enough controlling viruses and fruit flies. 

Among the cognoscenti of nanotechnology, this threat has become known as the "gray goo problem." 
Though masses of uncontrolled replicators need not be gray or gooey, the term "gray goo" emphasizes 
that replicators able to obliterate life might be less inspiring than a single species of crabgrass. They 
might be superior in an evolutionary sense, but this need not make them valuable. 

The gray goo threat makes one thing perfectly clear: We cannot afford certain kinds of accidents with 
replicating assemblers. 

Gray goo would surely be a depressing ending to our human adventure on Earth, far worse than mere 
fire or ice, and one that could stem from a simple laboratory accident.6 Oops. 

It is most of all the power of destructive self-replication in genetics, nanotechnology, and robotics 
(GNR) that should give us pause. Self-replication is the modus operandi of genetic engineering, which 
uses the machinery of the cell to replicate its designs, and the prime danger underlying gray goo in 
nanotechnology. Stories of run-amok robots like the Borg, replicating or mutating to escape from the 
ethical constraints imposed on them by their creators, are well established in our science fiction books 
and movies. It is even possible that self-replication may be more fundamental than we thought, and 



hence harder - or even impossible - to control. A recent article by Stuart Kauffman mNature titled 
"Self-Replication: Even Peptides Do It" discusses the discovery that a 32-amino-acid peptide can 
"autocatalyse its own synthesis." We don't know how widespread this ability is, but Kauffman notes 
that it may hint at "a route to self-reproducing molecular systems on a basis far wider than Watson- 
Crick base-pairing."? 

In truth, we have had in hand for years clear warnings of the dangers inherent in widespread knowledge 
of GNR technologies - of the possibility of knowledge alone enabling mass destruction. But these 
warnings haven't been widely publicized; the public discussions have been clearly inadequate. There is 
no profit in publicizing the dangers. 

The nuclear, biological, and chemical (NBC) technologies used in 20th-century weapons of mass 
destruction were and are largely military, developed in government laboratories. In sharp contrast, the 
2 1 st-century GNR technologies have clear commercial uses and are being developed almost 
exclusively by corporate enterprises. In this age of triumphant commercialism, technology - with 
science as its handmaiden - is delivering a series of almost magical inventions that are the most 
phenomenally lucrative ever seen. We are aggressively pursuing the promises of these new 
technologies within the now-unchallenged system of global capitalism and its manifold financial 
incentives and competitive pressures. 

This is the first moment in the history of our planet when any species, by its own voluntary actions, has 
become a danger to itself - as well as to vast numbers of others. 

It might be a familiar progression, transpiring on many worlds - a planet, newly formed, placidly 
revolves around its star; life slowly forms; a kaleidoscopic procession of creatures evolves; intelligence 
emerges which, at least up to a point, confers enormous survival value; and then technology is 
invented. It dawns on them that there are such things as laws of Nature, that these laws can be revealed 
by experiment, and that knowledge of these laws can be made both to save and to take lives, both on 
unprecedented scales. Science, they recognize, grants immense powers. In a fiash, they create world- 
altering contrivances. Some planetary civilizations see their way through, place limits on what may and 
what must not be done, and safely pass through the time of perils. Others, not so lucky or so prudent, 
perish. 



5 First Foresight Conference on Nanotechnology in October 1989, a talk titled "The Future of 
Computation." Published in Crandall, B. C. and James Lewis, editovs.Nanotechnology: Research and 
Perspectives. MIT Press, 1992: 269. See also www.foresight.org/Conferences/MNT01/Nanol.html . 

6 In his 1963 noYoiCat's Cradle, Kurt Vonnegut imagined a gray-goo-like accident where a form of ice 
called ice-nine, which becomes solid at a much higher temperature, freezes the oceans. 

7 Kauffman, Stuart. "Self-rephcation: Even Peptides Do It." Nature, 382, August 8, 1996: 496. 
See www.santafe.edu/sfi/People/kauffman/sak-peptides.html . 

That is Carl Sagan, writing in 1994, mPale Blue Dot, a book describing his vision of the human future 
in space. I am only now realizing how deep his insight was, and how sorely I miss, and will miss, his 
voice. For all its eloquence, Sagan's contribution was not least that of simple common sense - an 
attribute that, along with humility, many of the leading advocates of the 2 1 st-century technologies seem 
to lack. 

I remember from my childhood that my grandmother was strongly against the overuse of antibiotics. 



She had worked smce before the first World War as a nurse and had a commonsense attitude that taking 
antibiotics, unless they were absolutely necessary, was bad for you. 

It is not that she was an enemy of progress. She saw much progress in an almost 70-year nursing 
career; my grandfather, a diabetic, benefited greatly from the improved treatments that became 
available in his lifetime. But she, like many levelheaded people, would probably think it greatly 
arrogant for us, now, to be designing a robotic "replacement species," when we obviously have so much 
trouble making relatively simple things work, and so much trouble managing - or even understanding - 
ourselves. 

I realize now that she had an awareness of the nature of the order of life, and of the necessity of living 
with and respecting that order. With this respect comes a necessary humility that we, with our early- 
21st-century chutzpah, lack at our peril. The commonsense view, grounded in this respect, is often 
right, in advance of the scientific evidence. The clear fragility and inefficiencies of the human-made 
systems we have built should give us all pause; the fragility of the systems I have worked on certainly 
humbles me. 

We should have learned a lesson from the making of the first atomic bomb and the resulting arms race. 
We didn't do well then, and the parallels to our current situation are troubling. 

The effort to build the first atomic bomb was led by the brilliant physicist J. Robert Oppenheimer. 
Oppenheimer was not naturally interested in politics but became painfully aware of what he perceived 
as the grave threat to Western civilization from the Third Reich, a threat surely grave because of the 
possibility that Hitler might obtain nuclear weapons. Energized by this concern, he brought his strong 
intellect, passion for physics, and charismatic leadership skills to Los Alamos and led a rapid and 
successful effort by an incredible collection of great minds to quickly invent the bomb. 

What is striking is how this effort continued so naturally after the initial impetus was removed. In a 
meeting shortly after V-E Day with some physicists who felt that perhaps the effort should stop, 
Oppenheimer argued to continue. His stated reason seems a bit strange: not because of the fear of large 
casualties from an invasion of Japan, but because the United Nations, which was soon to be formed, 
should have foreknowledge of atomic weapons. A more likely reason the project continued is the 
momentum that had built up - the first atomic test. Trinity, was nearly at hand. 

We know that in preparing this first atomic test the physicists proceeded despite a large number of 
possible dangers. They were initially worried, based on a calculation by Edward Teller, that an atomic 
explosion might set fire to the atmosphere. A revised calculation reduced the danger of destrojdng the 
world to a three-in-a-million chance. (Teller says he was later able to dismiss the prospect of 
atmospheric ignition entirely.) Oppenheimer, though, was sufficiently concerned about the result of 
Trinity that he arranged for a possible evacuation of the southwest part of the state of New Mexico. 
And, of course, there was the clear danger of starting a nuclear arms race. 

Within a month of that first, successfiil test, two atomic bombs desfroyed Hiroshima and Nagasaki. 
Some scientists had suggested that the bomb simply be demonsfrated, rather than dropped on Japanese 
cities - saying that this would greatly improve the chances for arms control after the war - but to no 
avail. With the tragedy of Pearl Harbor still fresh in Americans' minds, it would have been very 
difficult for President Truman to order a demonstration of the weapons rather than use them as he did - 
the desire to quickly end the war and save the lives that would have been lost in any invasion of Japan 
was very strong. Yet the overriding truth was probably very simple: As the physicist Freeman Dyson 
later said, "The reason that it was dropped was just that nobody had the courage or the foresight to say 



It's important to realize how shocked the physicists were in the aftermath of the bombing of Hiroshima, 



on August 6, 1945. They describe a series of waves of emotion: first, a sense of fulfillment that the 
bomb worked, then horror at all the people that had been killed, and then a convincing feeling that on 
no account should another bomb be dropped. Yet of course another bomb was dropped, on Nagasaki, 
only three days after the bombing of Hiroshima. 

In November 1945, three months after the atomic bombings, Oppenheimer stood firmly behind the 
scientific attitude, saying, "It is not possible to be a scientist unless you believe that the knowledge of 
the world, and the power which this gives, is a thing which is of intrinsic value to humanity, and that 
you are using it to help in the spread of knowledge and are willing to take the consequences." 

Oppenheimer went on to work, with others, on the Acheson-Lilienthal report, which, as Richard 
Rhodes says in his recent hookVisions of Technology, "found a way to prevent a clandestine nuclear 
arms race without resorting to armed world government"; their suggestion was a form of 
relinquishment of nuclear weapons work by nation-states to an international agency. 

This proposal led to the Baruch Plan, which was submitted to the United Nations in June 1946 but 
never adopted (perhaps because, as Rhodes suggests, Bernard Baruch had "insisted on burdening the 
plan with conventional sanctions," thereby inevitably dooming it, even though it would "almost 
certainly have been rejected by Stalinist Russia anyway"). Other efforts to promote sensible steps 
toward internationalizing nuclear power to prevent an arms race ran afoul either of US politics and 
internal distrust, or distrust by the Soviets. The opportunity to avoid the arms race was lost, and very 
quickly. 

Two years later, in 1948, Oppenheimer seemed to have reached another stage in his thinking, saying, 
"In some sort of crude sense which no vulgarity, no humor, no overstatement can quite extinguish, the 
physicists have known sin; and this is a knowledge they cannot lose." 

In 1949, the Soviets exploded an atom bomb. By 1955, both the US and the Soviet Union had tested 
hydrogen bombs suitable for delivery by aircraft. And so the nuclear arms race began. 

Nearly 20 years ago, in the documentary 7% e Day After Trinity, Freeman Dyson summarized the 
scientific attitudes that brought us to the nuclear precipice: 

"I have felt it myself. The glitter of nuclear weapons. It is irresistible if you come to them as a scientist. 
To feel it's there in your hands, to release this energy that fuels the stars, to let it do your bidding. To 
perform these miracles, to lift a million tons of rock into the sky. It is something that gives people an 
illusion of illimitable power, and it is, in some ways, responsible for all our troubles - this, what you 
might call technical arrogance, that overcomes people when they see what they can do with their 
minds. "8 

Now, as then, we are creators of new technologies and stars of the imagined future, driven - this time 
by great financial rewards and global competition - despite the clear dangers, hardly evaluating what it 
may be like to try to live in a world that is the realistic outcome of what we are creating and imagining. 



In 1947, The Bulletin of the Atomic Scientists began putting a Doomsday Clock on its cover. For more 
than 50 years, it has shown an estimate of the relative nuclear danger we have faced, reflecting the 
changing international conditions. The hands on the clock have moved 15 times and today, standing at 
nine minutes to midnight, reflect continuing and real danger from nuclear weapons. The recent addition 
of India and Pakistan to the list of nuclear powers has increased the threat of failure of the 
nonproliferation goal, and this danger was reflected by moving the hands closer to midnight in 1998. 

In our time, how much danger do we face, not just from nuclear weapons, but from all of these 
technologies? How high are the extinction risks? 



The philosopher John Leslie has studied this question and concluded that the risk of human extinction 
is at least 30 percent,9 while Ray Kurzweil believes we have "a better than even chance of making it 
through," with the caveat that he has "always been accused of being an optimist." Not only are these 
estimates not encouraging, but they do not include the probability of many horrid outcomes that lie 
short of extinction. 

Faced with such assessments, some serious people are already suggesting that we simply move beyond 
Earth as quickly as possible. We would colonize the galaxy using von Neumann probes, which hop 
from star system to star system, replicating as they go. This step will almost certainly be necessary 5 
billion years from now (or sooner if our solar system is disastrously impacted by the impending 
collision of our galaxy with the Andromeda galaxy within the next 3 billion years), but if we take 
Kurzweil and Moravec at their word it might be necessary by the middle of this century. 

What are the moral implications here? If we must move beyond Earth this quickly in order for the 
species to survive, who accepts the responsibility for the fate of those (most of us, after all) who are left 
behind? And even if we scatter to the stars, isn't it likely that we may take our problems with us or find, 
later, that they have followed us? The fate of our species on Earth and our fate in the galaxy seem 
inextricably linked. 

Another idea is to erect a series of shields to defend against each of the dangerous technologies. The 
Strategic Defense Initiative, proposed by the Reagan administration, was an attempt to design such a 
shield against the threat of a nuclear attack from the Soviet Union. But as Arthur C. Clarke, who was 
privy to discussions about the project, observed: "Though it might be possible, at vast expense, to 
construct local defense systems that would 'only' let through a few percent of ballistic missiles, the 
much touted idea of a national umbrella was nonsense. Luis Alvarez, perhaps the greatest experimental 
physicist of this century, remarked to me that the advocates of such schemes were 'very bright guys 
with no common sense.'" 

Clarke continued: "Looking into my often cloudy crystal ball, I suspect that a total defense might 
indeed be possible in a century or so. But the technology involved would produce, as a by-product, 
weapons so terrible that no one would bother with anything as primitive as ballistic missiles." 10 

InEngines of Creation, Eric Drexler proposed that we build an active nanotechnological shield - a form 
of immune system for the biosphere - to defend against dangerous replicators of all kinds that might 
escape from laboratories or otherwise be maliciously created. But the shield he proposed would itself 
be extremely dangerous - nothing could prevent it from developing autoimmune problems and 
attacking the biosphere itself JJ. 

Similar difficulties apply to the construction of shields against robotics and genetic engineering. These 
technologies are too powerful to be shielded against in the time frame of interest; even if it were 
possible to implement defensive shields, the side effects of their development would be at least as 
dangerous as the technologies we are trying to protect against. 

These possibilities are all thus either undesirable or unachievable or both. The only realistic alternative 
I see is relinquishment: to limit development of the technologies that are too dangerous, by limiting our 
pursuit of certain kinds of knowledge. 

Yes, I know, knowledge is good, as is the search for new truths. We have been seeking knowledge since 
ancient times. Aristotle opened his Metaphysics with the simple statement: "All men by nature desire to 
know." We have, as a bedrock value in our society, long agreed on the value of open access to 
information, and recognize the problems that arise with attempts to restrict access to and development 
of knowledge. In recent times, we have come to revere scientific knowledge. 



8 Else, Jon.The Day After Trinity: J. Robert Oppenheimer and The Atomic Bomb (available at 
www.pyramiddirect.com ). 

9 This estimate is in Leslie's hookThe End of the World: The Science and Ethics of Human Extinction, 
where he notes that the probability of extinction is substantially higher if we accept Brandon Carter's 
Doomsday Argument, which is, briefly, that "we ought to have some reluctance to believe that we are 
very exceptionally early, for instance in the earliest 0.001 percent, among all humans who will ever 
have lived. This would be some reason for thinking that humankind will not survive for many more 
centuries, let alone colonize the galaxy. Carter's doomsday argument doesn't generate any risk estimates 
just by itself It is an argument foxrevising the estimates which we generate when we consider various 
possible dangers." (Routledge, 1996: 1, 3, 145.) 

10 Clarke, Arthur C. "Presidents, Experts, and Asteroids. "^c/ence, June 5, 1998. Reprinted as "Science 
and Society" mGreetings, Carbon-Based Bipeds! Collected Essays, 1934-1998. St. Martin's Press, 
1999: 526. 

11 And, as David Forrest suggests in his paper "Regulating Nanotechnology Development," available 
at www.foresight.org/NanoRev/Forrestl989.html . "If we used strict liability as an alternative to 
regulation it would be impossible for any developer to internalize the cost of the risk (destruction of the 
biosphere), so theoretically the activity of developing nanotechnology should never be undertaken." 
Forrest's analysis leaves us with only government regulation to protect us - not a comforting thought. 

But despite the strong historical precedents, if open access to and unlimited development of knowledge 
henceforth puts us all in clear danger of extinction, then common sense demands that we reexamine 
even these basic, long-held behefs. 

It was Nietzsche who warned us, at the end of the 19th century, not only that God is dead but that "faith 
in science, which after all exists undeniably, cannot owe its origin to a calculus of utility; it must have 
originated in spite of the fact that the disutility and dangerousness of the 'will to truth,' of 'truth at any 
price' is proved to it constantly." It is this further danger that we now fully face - the consequences of 
our truth-seeking. The truth that science seeks can certainly be considered a dangerous substitute for 
God if it is likely to lead to our extinction. 

If we could agree, as a species, what we wanted, where we were headed, and why, then we would make 
our future much less dangerous - then we might understand what we can and should relinquish. 
Otherwise, we can easily imagine an arms race developing over GNR technologies, as it did with the 
NBC technologies in the 20th century. This is perhaps the greatest risk, for once such a race begins, it's 
very hard to end it. This time - unlike during the Manhattan Project - we aren't in a war, facing an 
implacable enemy that is threatening our civilization; we are driven, instead, by our habits, our desires, 
our economic system, and our competitive need to know. 

I believe that we all wish our course could be determined by our collective values, ethics, and morals. 
If we had gained more collective wisdom over the past few thousand years, then a dialogue to this end 
would be more practical, and the incredible powers we are about to unleash would not be nearly so 
troubling. 

One would think we might be driven to such a dialogue by our instinct for self-preservation. 
Individuals clearly have this desire, yet as a species our behavior seems to be not in our favor. In 
dealing with the nuclear threat, we often spoke dishonestly to ourselves and to each other, thereby 
greatly increasing the risks. Whether this was politically motivated, or because we chose not to think 
ahead, or because when faced with such grave threats we acted irrationally out of fear, I do not know, 
but it does not bode well. 

The new Pandora's boxes of genetics, nanotechnology, and robotics are almost open, yet we seem 



hardly to have noticed. Ideas can't be put back in a box; unlike uranium or plutonium, they don't need 
to be mined and refined, and they can be fi-eely copied. Once they are out, they are out. Churchill 
remarked, in a famous left-handed compliment, that the American people and their leaders "invariably 
do the right thing, after they have examined every other alternative." In this case, however, we must act 
more presciently, as to do the right thing only at last may be to lose the chance to do it at all. 

As Thoreau said, "We do not ride on the railroad; it rides upon us"; and this is what we must fight, in 
our time. The question is, indeed. Which is to be master? Will we survive our technologies? 

We are being propelled into this new century with no plan, no control, no brakes. Have we already gone 
too far down the path to alter course? I don't believe so, but we aren't trying yet, and the last chance to 
assert control - the fail-safe point - is rapidly approaching. We have our first pet robots, as well as 
commercially available genetic engineering techniques, and our nanoscale techniques are advancing 
rapidly. While the development of these technologies proceeds through a number of steps, it isn't 
necessarily the case - as happened in the Manhattan Project and the Trinity test - that the last step in 
proving a technology is large and hard. The breakthrough to wild self-replication in robotics, genetic 
engineering, or nanotechnology could come suddenly, reprising the surprise we felt when we learned of 
the cloning of a mammal. 

And yet I believe we do have a strong and solid basis for hope. Our attempts to deal with weapons of 
mass destruction in the last century provide a shining example of relinquishment for us to consider: the 
unilateral US abandonment, without preconditions, of the development of biological weapons. This 
relinquishment stemmed from the realization that while it would take an enormous effort to create these 
terrible weapons, they could from then on easily be duplicated and fall into the hands of rogue nations 
or terrorist groups. 

The clear conclusion was that we would create additional threats to ourselves by pursuing these 
weapons, and that we would be more secure if we did not pursue them. We have embodied our 
relinquishment of biological and chemical weapons in the 1972 Biological Weapons Convention 
(BWC) and the 1993 Chemical Weapons Convention (CWC).I2 

As for the continuing sizable threat from nuclear weapons, which we have lived with now for more 
than 50 years, the US Senate's recent rejection of the Comprehensive Test Ban Treaty makes it clear 
relinquishing nuclear weapons will not be politically easy. But we have a unique opportunity, with the 
end of the Cold War, to avert a multipolar arms race. Building on the BWC and CWC relinquishments, 
successful abolition of nuclear weapons could help us build toward a habit of relinquishing dangerous 
technologies. (Actually, by getting rid of all but 100 nuclear weapons worldwide - roughly the total 
destructive power of World War II and a considerably easier task - we could eliminate this extinction 
threat. 13) 

Verifying relinquishment will be a difficult problem, but not an unsolvable one. We are fortunate to 
have already done a lot of relevant work in the context of the BWC and other treaties. Our major task 
will be to apply this to technologies that are naturally much more commercial than military. The 
substantial need here is for transparency, as difficulty of verification is directly proportional to the 
difficulty of distinguishing relinquished from legitimate activities. 



12 Meselson, Matthew. "The Problem of Biological Weapons." Presentation to the 1,818th Stated 
Meeting of the American Academy of Arts and Sciences, January 13, 1999. 
( minerva. amacad. org/ archive/bulletin4 . htm ) 



13 Doty, Paul. "The Forgotten Menace: Nuclear Weapons Stockpiles Still Represent the Biggest Threat 
to Civilization. 'Wa^Mre, 402, December 9, 1999: 583. 

I frankly believe that the situation in 1945 was simpler than the one we now face: The nuclear 
technologies were reasonably separable into commercial and military uses, and monitoring was aided 
by the nature of atomic tests and the ease with which radioactivity could be measured. Research on 
military applications could be performed at national laboratories such as Los Alamos, with the results 
kept secret as long as possible. 

The GNR technologies do not divide clearly into commercial and military uses; given their potential in 
the market, it's hard to imagine pursuing them only in national laboratories. With their widespread 
commercial pursuit, enforcing relinquishment will require a verification regime similar to that for 
biological weapons, but on an unprecedented scale. This, inevitably, will raise tensions between our 
individual privacy and desire for proprietary information, and the need for verification to protect us all. 
We will undoubtedly encounter strong resistance to this loss of privacy and freedom of action. 

Verifying the relinquishment of certain GNR technologies will have to occur in cyberspace as well as at 
physical facilities. The critical issue will be to make the necessary transparency acceptable in a world 
of proprietary information, presumably by providing new forms of protection for intellectual property. 

Verifying compliance will also require that scientists and engineers adopt a strong code of ethical 
conduct, resembling the Hippocratic oath, and that they have the courage to whistleblow as necessary, 
even at high personal cost. This would answer the call - 50 years after Hiroshima - by the Nobel 
laureate Hans Bethe, one of the most senior of the surviving members of the Manhattan Project, that all 
scientists "cease and desist from work creating, developing, improving, and manufacturing nuclear 
weapons and other weapons of potential mass destruction. "14 In the 21st century, this requires 
vigilance and personal responsibility by those who would work on both NBC and GNR technologies to 
avoid implementing weapons of mass destruction and knowledge-enabled mass destruction. 

Thoreau also said that we will be "rich in proportion to the number of things which we can afford to let 
alone." We each seek to be happy, but it would seem worthwhile to question whether we need to take 
such a high risk of total destruction to gain yet more knowledge and yet more things; common sense 
says that there is a limit to our material needs - and that certain knowledge is too dangerous and is best 
forgone. 

Neither should we pursue near immortality without considering the costs, without considering the 
commensurate increase in the risk of extinction. Immortality, while perhaps the original, is certainly not 
the only possible Utopian dream. 

I recently had the good fortune to meet the distinguished author and scholar Jacques Attali, whose 
hookLignes d'horizons {Millennium, in the English translation) helped inspire the Java and Jini 
approach to the coming age of pervasive computing, as previously described in this magazine. In his 
new hodkFraternites, Attali describes how our dreams of Utopia have changed over time: 

"At the dawn of societies, men saw their passage on Earth as nothing more than a labyrinth of pain, at 
the end of which stood a door leading, via their death, to the company of gods and ioEternity. With the 
Hebrews and then the Greeks, some men dared free themselves from theological demands and dream of 
an ideal City whereLiberty would flourish. Others, noting the evolution of the market society, 
understood that the liberty of some would entail the alienation of others, and they soughtEquality." 

Jacques helped me understand how these three different Utopian goals exist in tension in our society 
today. He goes on to describe a fourth utopia,Fratemity, whose foundation is altruism. Fraternity alone 



associates individual happiness with the happiness of others, affording the promise of self-sustainment. 

This crystalUzed for me my problem with Kurzweil's dream. A technological approach to Eternity - 
near immortality through robotics - may not be the most desirable Utopia, and its pursuit brings clear 
dangers. Maybe we should rethink our Utopian choices. 

Where can we look for a new ethical basis to set our course? I have found the ideas in the book Ethics 
for the New Millennium, by the Dalai Lama, to be very helpful. As is perhaps well known but little 
heeded, the Dalai Lama argues that the most important thing is for us to conduct our lives with love and 
compassion for others, and that our societies need to develop a stronger notion of universal 
responsibility and of our interdependency; he proposes a standard of positive ethical conduct for 
individuals and societies that seems consonant with Attali's Fraternity utopia. 

The Dalai Lama further argues that we must understand what it is that makes people happy, and 
acknowledge the strong evidence that neither material progress nor the pursuit of the power of 
knowledge is the key - that there are limits to what science and the scientific pursuit alone can do. 

Our Western notion of happiness seems to come from the Greeks, who defined it as "the exercise of 
vital powers along lines of excellence in a life affording them scope." 15 

Clearly, we need to find meaningful challenges and sufficient scope in our lives if we are to be happy in 
whatever is to come. But I believe we must find alternative outlets for our creative forces, beyond the 
culture of perpetual economic growth; this growth has largely been a blessing for several hundred 
years, but it has not brought us unalloyed happiness, and we must now choose between the pursuit of 
unrestricted and undirected growth through science and technology and the clear accompanying 
dangers. 

It is now more than a year since my first encounter with Ray Kurzweil and John Searle. I see around 
me cause for hope in the voices for caution and relinquishment and in those people I have discovered 
who are as concerned as I am about our current predicament. I feel, too, a deepened sense of personal 
responsibility - not for the work I have already done, but for the work that I might yet do, at the 
confluence of the sciences. 

14 See also Hans Bethe's 1997 letter to President Clinton, at www. fas . org/bethecr.htm . 

15 Hamilton, Edith.rAe Greek Way. W. W. Norton & Co., 1942: 35. 

SOURCE http://archive.wired.eom/wired/archive/8.04/joy.html 

US Boston Dynamics - Petman Robot (Darpa) VIDEO BELOW 
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